{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with TensorFlow (Graph Mode)\n",
    "\n",
    "**Learning Objectives**\n",
    "  - Understand the difference between Tensorflow's two modes: Eager Execution and Graph Execution\n",
    "  - Get used to deferred execution paradigm: first define a graph then run it in a `tf.Session()`\n",
    "  - Understand how to parameterize a graph using `tf.placeholder()` and `feed_dict`\n",
    "  - Understand the difference between constant Tensors and variable Tensors, and how to define each\n",
    "  - Practice using mid-level `tf.train` module for gradient descent\n",
    "\n",
    "## Introduction\n",
    "\n",
    "**Eager Execution**\n",
    "\n",
    "Eager mode evaluates operations immediatley and return concrete values immediately. To enable eager mode simply place `tf.enable_eager_execution()` at the top of your code. We recommend using eager execution when prototyping as it is intuitive, easier to debug, and requires less boilerplate code.\n",
    "\n",
    "**Graph Execution**\n",
    "\n",
    "Graph mode is TensorFlow's default execution mode (although it will change to eager in TF 2.0). In graph mode operations only produce a symbolic graph which doesn't get executed until run within the context of a tf.Session(). This style of coding is less inutitive and has more boilerplate, however it can lead to performance optimizations and is particularly suited for distributing training across multiple devices. We recommend using delayed execution for performance sensitive production code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graph Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Two Tensors "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build the Graph\n",
    "\n",
    "Unlike eager mode, no concrete value will be returned yet. Just a name, shape and type are printed. Behind the scenes a directed graph is being created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.constant(value = [5, 3, 8], dtype = tf.int32)\n",
    "b = tf.constant(value = [3, -1, 2], dtype = tf.int32)\n",
    "c = tf.add(x = a, y = b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Graph\n",
    "\n",
    "A graph can be executed in the context of a `tf.Session()`. Think of a session as the bridge between the front-end Python API and the back-end C++ execution engine. \n",
    "\n",
    "Within a session, passing a tensor operation to `run()` will cause Tensorflow to execute all upstream operations in the graph required to calculate that value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    result = sess.run(fetches = c)\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parameterizing the Grpah \n",
    "\n",
    "What if values of `a` and `b` keep changing? How can you parameterize them so they can be fed in at runtime? \n",
    "\n",
    "*Step 1: Define Placeholders*\n",
    "\n",
    "Define `a` and `b` using `tf.placeholder()`. You'll need to specify the data type of the placeholder, and optionally a tensor shape.\n",
    "\n",
    "*Step 2: Provide feed_dict*\n",
    "\n",
    "Now when invoking `run()` within the `tf.Session()`, in addition to providing a tensor operation to evaluate, you also provide a dictionary whose keys are the names of the placeholders. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.placeholder(dtype = tf.int32, shape = [None])  \n",
    "b = tf.placeholder(dtype = tf.int32, shape = [None])\n",
    "c = tf.add(x = a, y = b)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    result = sess.run(fetches = c, feed_dict = {\n",
    "        a: [3, 4, 5],\n",
    "        b: [-1, 2, 3]\n",
    "    })\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Toy Dataset\n",
    "We'll model the following:\n",
    "\n",
    "\\begin{equation}\n",
    "y= 2x + 10\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.constant(value = [1,2,3,4,5,6,7,8,9,10], dtype = tf.float32)\n",
    "Y = 2 * X + 10\n",
    "print(\"X:{}\".format(X))\n",
    "print(\"Y:{}\".format(Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Loss Function\n",
    "Using mean squared error, our loss function is:\n",
    "\\begin{equation}\n",
    "MSE = \\frac{1}{m}\\sum_{i=1}^{m}(\\hat{Y}_i-Y_i)^2\n",
    "\\end{equation}\n",
    "\n",
    "$\\hat{Y}$ represents the vector containing our model's predictions:\n",
    "\\begin{equation}\n",
    "\\hat{Y} = w_0X + w_1\n",
    "\\end{equation}\n",
    "\n",
    "Note below we introduce TF variables for the first time. Unlike constants, variables are mutable. \n",
    "\n",
    "Browse the official TensorFlow [guide on variables](https://www.tensorflow.org/guide/variables) for more information on when/how to use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope(name_or_scope = \"training\", reuse = tf.AUTO_REUSE):\n",
    "    w0 = tf.get_variable(name = \"w0\", initializer = tf.constant(value = 0.0, dtype = tf.float32))\n",
    "    w1 = tf.get_variable(name = \"w1\", initializer = tf.constant(value = 0.0, dtype = tf.float32))\n",
    "    \n",
    "Y_hat = w0 * X + w1\n",
    "loss_mse = tf.reduce_mean(input_tensor = (Y_hat - Y)**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer\n",
    "\n",
    "An optimizer in TensorFlow both calculates gradients and updates weights. In addition to basic  gradient descent, TF provides implementations of several more advanced optimizers such as ADAM and FTRL. They can all be found in the [tf.train](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/train) module. \n",
    "\n",
    "Note below we're not expclictly telling the optimizer which tensors are our weight tensors. So how does it know what to update? Optimizers will update all variables in the `tf.GraphKeys.TRAINABLE_VARIABLES` [collection](https://www.tensorflow.org/guide/variables#variable_collections). All variables are added to this collection by default. Since our only variables are `w0` and `w1`, this is the behavior we want. If we had a variable that we *didn't* want to be added to the collection we would set `trainable=false` when creating it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = tf.placeholder(dtype = tf.float32, shape = None)\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate = LEARNING_RATE).minimize(loss = loss_mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Loop\n",
    "\n",
    "Note our results are identical to what we found in Eager mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "STEPS = 1000\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer()) # initialize variables\n",
    "    \n",
    "    for step in range(STEPS):\n",
    "        #1. Calculate gradients and update seights \n",
    "        sess.run(fetches = optimizer, feed_dict = {LEARNING_RATE: 0.02})\n",
    "        \n",
    "        #2. Periodically print MSE\n",
    "        if step % 100 == 0:\n",
    "            print(\"STEP: {} MSE: {}\".format(step, sess.run(fetches = loss_mse)))\n",
    "    \n",
    "    # Print final MSE and weights\n",
    "    print(\"STEP: {} MSE: {}\".format(STEPS, sess.run(loss_mse)))\n",
    "    print(\"w0:{}\".format(round(float(sess.run(w0)), 4)))\n",
    "    print(\"w1:{}\".format(round(float(sess.run(w1)), 4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
